Determining an Image Capture Payload Burst Structure Based on a Metering Image Capture Sweep

ABSTRACT

A first plurality of images of a scene may be captured. Each image of the first plurality of images may be captured with a different total exposure time (TET). Based at least on the first plurality of images, a TET sequence may be determined for capturing images of the scene. A second plurality of images of the scene may be captured. Images in the second plurality of images may be captured using the TET sequence. Based at least on the second plurality of images, an output image of the scene may be constructed.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.13/713,720, filed Dec. 13, 2012, which is hereby incorporated byreference in its entirety, and to which this application claimspriority.

BACKGROUND

Generally, imaging may refer to representing the color and brightnesscharacteristics of digital images. Low dynamic range (LDR) imaging mayrepresent digital images (e.g., photographs and motion video) with 8 orfewer bits for each color channel of a pixel. As a result, up to 256levels of brightness may be supported. Currently, a wide range of videooutput devices (e.g., computer monitors, tablet and smartphone screens,televisions, etc.) support displaying LDR images.

However, real-world scenes often exhibit a wider range of brightnessthan can be represented by LDR imaging. As an example scene with a widebrightness range, consider an individual standing in a dark room infront of a window. This scene may include both extremely bright regions(e.g., sunlit features outside the window) and extremely dark regions(e.g., the features in the room). Ideally, a photograph of this scenewould include both the details in the bright regions and the details inthe dark regions.

SUMMARY

In a first example embodiment, a first plurality of images of a scenemay be captured. Each image of the first plurality of images may becaptured with a different total exposure time (TET). Based at least onthe first plurality of images, a TET sequence may be determined forcapturing further images of the scene. A second plurality of images ofthe scene may be captured. Images in the second plurality of images maybe captured using the TET sequence. Based at least on the secondplurality of images, an output image of the scene may be constructed.

In a second example embodiment, a first plurality of images of a scenemay be captured. Each image of the first plurality of images may becaptured using a different TET. Based at least on the first plurality ofimages, a long TET, a short TET, and a TET sequence that includes thelong TET and the short TET may be determined. A second plurality ofimages of the scene may be captured. The images in the second pluralityof images may be captured sequentially in an image sequence using asequence of TETs corresponding to the TET sequence. Based on one or moreimages in the image sequence, an output image may be constructed.

A third example embodiment may include a non-transitorycomputer-readable storage medium, having stored thereon programinstructions that, upon execution by a computing device, cause thecomputing device, and/or its peripherals, to perform operations inaccordance with the first and/or second example embodiments.

A fourth example embodiment may include a computing device, comprisingat least a processor, an image sensor, and data storage. The datastorage may contain program instructions that, upon execution by theprocessor, cause the computing device operate in accordance with thefirst and/or second example embodiments.

These as well as other aspects, advantages, and alternatives will becomeapparent to those of ordinary skill in the art by reading the followingdetailed description with reference where appropriate to theaccompanying drawings. Further, it should be understood that thedescription provided in this summary section and elsewhere in thisdocument is intended to illustrate the claimed subject matter by way ofexample and not by way of limitation.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 depicts front, right side, and rear views of a digital cameradevice, in accordance with an example embodiment.

FIG. 2 depicts a block diagram of a computing device with image capturecapability, in accordance with an example embodiment.

FIG. 3 depicts a flow chart, in accordance with an example embodiment.

FIG. 4A depicts creating a histogram from one or more captured images,in accordance with an example embodiment.

FIG. 4B depicts a training image database, in accordance with an exampleembodiment.

FIG. 5 depicts payload burst structures for capturing images, inaccordance with an example embodiment.

FIG. 6 depicts a flow chart, in accordance with an example embodiment.

FIG. 7 depicts another flow chart, in accordance with an exampleembodiment.

DETAILED DESCRIPTION

As image capture devices, such as cameras, become more popular, they maybe employed as standalone hardware devices or integrated into variousother types of devices. For instance, still and video cameras are nowregularly included in wireless communication devices (e.g., mobilephones), tablet computers, laptop computers, video game interfaces, homeautomation devices, and even automobiles and other types of vehicles.

The physical components of a camera may include an aperture throughwhich light enters, a recording surface for capturing the imagerepresented by the light, and a lens positioned in front of the apertureto focus at least part of the image on the recording surface. Theaperture may be fixed size or adjustable. In an analog camera, therecording surface may be photographic film. In a digital camera, therecording surface may include an electronic image sensor (e.g., a chargecoupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS)sensor) to transfer and/or store captured images in a data storage unit(e.g., memory).

A shutter may be coupled to or nearby the lens or the recording surface.The shutter may either be in a closed position, in which it blocks lightfrom reaching the recording surface, or an open position, in which lightis allowed to reach to recording surface. The position of the shuttermay be controlled by a shutter button. For instance, the shutter may bein the closed position by default. When the shutter button is triggered(e.g., pressed), the shutter may change from the closed position to theopen position for a period of time, known as the shutter cycle. Duringthe shutter cycle, an image may be captured on the recording surface. Atthe end of the shutter cycle, the shutter may change back to the closedposition.

Alternatively, the shuttering process may be electronic. For example,before an electronic shutter of a CCD image sensor is “opened” thesensor may be reset to remove any residual signal in its photodiodes.While the electronic shutter remains open, the photodiodes mayaccumulate charge. When or after the shutter closes, these charges maybe transferred to longer-term data storage. Combinations of mechanicaland electronic shuttering may also be possible.

Regardless of type, a shutter may be activated and/or controlled bysomething other than a shutter button. For instance, the shutter may beactivated by a softkey, a timer, or some other trigger. Herein, the term“image capture” may refer to any mechanical and/or electronic shutteringprocess that results in one or more photographs being recorded,regardless of how the shuttering process is triggered or controlled.

The exposure of a captured image may be determined by a combination ofthe size of the aperture, the brightness of the light entering theaperture, and the length of the shutter cycle (also referred to as theshutter length or the exposure length). Additionally, a digital oranalog gain may be applied to the image, thereby influencing theexposure. In some embodiments, the term “total exposure length” or“total exposure time” may refer to the shutter length multiplied by thegain for a particular aperture size. Herein, the term “total exposuretime,” or “TET,” should be interpreted as possibly being a shutterlength, an exposure time, or any other metric that controls the amountof signal response that results from light reaching the recordingsurface.

A still camera may capture one or more images each time image capture istriggered. A video camera may continuously capture images at aparticular rate (e.g., 24 images—or frames—per second) as long as imagecapture remains triggered (e.g., while the shutter button is held down).Some digital still cameras may open the shutter when the camera deviceor application is activated, and the shutter may remain in this positionuntil the camera device or application is deactivated. While the shutteris open, the camera device or application may capture and display arepresentation of a scene on a viewfinder. When image capture istriggered, one or more distinct digital images of the current scene maybe captured.

Cameras—even analog cameras—may include software to control one or morecamera functions and/or settings, such as aperture size, TET, gain, andso on. Additionally, some cameras may include software that digitallyprocesses images during or after these images are captured. While itshould be understood that the description above refers to cameras ingeneral, it may be particularly relevant to digital cameras.

As noted previously, digital cameras may be standalone devices orintegrated with other devices. As an example, FIG. 1 illustrates theform factor of a digital camera device 100. Digital camera device 100may be, for example, a mobile phone, a tablet computer, or a wearablecomputing device. However, other embodiments are possible. Digitalcamera device 100 may include various elements, such as a body 102, afront-facing camera 104, a multi-element display 106, a shutter button108, and other buttons 110. Digital camera device 100 could furtherinclude a rear-facing camera 112. Front-facing camera 104 may bepositioned on a side of body 102 typically facing a user while inoperation, or on the same side as multi-element display 106. Rear-facingcamera 112 may be positioned on a side of body 102 opposite front-facingcamera 104. Referring to the cameras as front and rear facing isarbitrary, and digital camera device 100 may include multiple cameraspositioned on various sides of body 102.

Multi-element display 106 could represent a cathode ray tube (CRT)display, a light emitting diode (LED) display, a liquid crystal (LCD)display, a plasma display, or any other type of display known in theart. In some embodiments, multi-element display 106 may display adigital representation of the current image being captured byfront-facing camera 104 and/or rear-facing camera 112, or an image thatcould be captured or was recently captured by either or both of thesecameras. Thus, multi-element display 106 may serve as a viewfinder foreither camera. Multi-element display 106 may also support touchscreenand/or presence-sensitive functions that may be able to adjust thesettings and/or configuration of any aspect of digital camera device100.

Front-facing camera 104 may include an image sensor and associatedoptical elements such as lenses. Front-facing camera 104 may offer zoomcapabilities or could have a fixed focal length. In other embodiments,interchangeable lenses could be used with front-facing camera 104.Front-facing camera 104 may have a variable mechanical aperture and amechanical and/or electronic shutter. Front-facing camera 104 also couldbe configured to capture still images, video images, or both. Further,front-facing camera 104 could represent a monoscopic, stereoscopic, ormultiscopic camera. Rear-facing camera 112 may be similarly ordifferently arranged. Additionally, front-facing camera 104, rear-facingcamera 112, or both, may be an array of one or more cameras.

Either or both of front facing camera 104 and rear-facing camera 112 mayinclude or be associated with an illumination component that provides alight field to illuminate a target object. For instance, an illuminationcomponent could provide flash or constant illumination of the targetobject. An illumination component could also be configured to provide alight field that includes one or more of structured light, polarizedlight, and light with specific spectral content. Other types of lightfields known and used to recover three-dimensional (3D) models from anobject are possible within the context of the embodiments herein.

Either or both of front facing camera 104 and rear-facing camera 112 mayinclude or be associated with an ambient light sensor that maycontinuously or from time to time determine the ambient brightness of ascene that the camera can capture. In some devices, the ambient lightsensor can be used to adjust the display brightness of a screenassociated with the camera (e.g., a viewfinder). When the determinedambient brightness is high, the brightness level of the screen may beincreased to make the screen easier to view. When the determined ambientbrightness is low, the brightness level of the screen may be decreased,also to make the screen easier to view as well as to potentially savepower. Additionally, the ambient light sensor's input may be used todetermine a TET of an associated camera, or to help in thisdetermination.

Digital camera device 100 could be configured to use multi-elementdisplay 106 and either front-facing camera 104 or rear-facing camera 112to capture images of a target object. The captured images could be aplurality of still images or a video stream. The image capture could betriggered by activating shutter button 108, pressing a softkey onmulti-element display 106, or by some other mechanism. Depending uponthe implementation, the images could be captured automatically at aspecific time interval, for example, upon pressing shutter button 108,upon appropriate lighting conditions of the target object, upon movingdigital camera device 100 a predetermined distance, or according to apredetermined capture schedule.

As noted above, the functions of digital camera device 100—or anothertype of digital camera—may be integrated into a computing device, suchas a wireless communication device, tablet computer, laptop computer andso on. For purposes of example, FIG. 2 is a simplified block diagramshowing some of the components of an example computing device 200 thatmay include camera components 224.

By way of example and without limitation, computing device 200 may be acellular mobile telephone (e.g., a smartphone), a still camera, a videocamera, a fax machine, a computer (such as a desktop, notebook, tablet,or handheld computer), a personal digital assistant (PDA), a homeautomation component, a digital video recorder (DVR), a digitaltelevision, a remote control, a wearable computing device, or some othertype of device equipped with at least some image capture and/or imageprocessing capabilities. It should be understood that computing device200 may represent a physical camera device such as a digital camera, aparticular physical hardware platform on which a camera applicationoperates in software, or other combinations of hardware and softwarethat are configured to carry out camera functions.

As shown in FIG. 2, computing device 200 may include a communicationinterface 202, a user interface 204, a processor 206, data storage 208,and camera components 224, all of which may be communicatively linkedtogether by a system bus, network, or other connection mechanism 210.

Communication interface 202 may function to allow computing device 200to communicate, using analog or digital modulation, with other devices,access networks, and/or transport networks. Thus, communicationinterface 202 may facilitate circuit-switched and/or packet-switchedcommunication, such as plain old telephone service (POTS) communicationand/or Internet protocol (IP) or other packetized communication. Forinstance, communication interface 202 may include a chipset and antennaarranged for wireless communication with a radio access network or anaccess point. Also, communication interface 202 may take the form of orinclude a wireline interface, such as an Ethernet, Universal Serial Bus(USB), or High-Definition Multimedia Interface (HDMI) port.Communication interface 202 may also take the form of or include awireless interface, such as a Wifi, BLUETOOTH®, global positioningsystem (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPPLong-Term Evolution (LTE)). However, other forms of physical layerinterfaces and other types of standard or proprietary communicationprotocols may be used over communication interface 202. Furthermore,communication interface 202 may comprise multiple physical communicationinterfaces (e.g., a Wifi interface, a BLUETOOTH® interface, and awide-area wireless interface).

User interface 204 may function to allow computing device 200 tointeract with a human or non-human user, such as to receive input from auser and to provide output to the user. Thus, user interface 204 mayinclude input components such as a keypad, keyboard, touch-sensitive orpresence-sensitive panel, computer mouse, trackball, joystick,microphone, and so on. User interface 204 may also include one or moreoutput components such as a display screen which, for example, may becombined with a presence-sensitive panel. The display screen may bebased on CRT, LCD, and/or LED technologies, or other technologies nowknown or later developed. User interface 204 may also be configured togenerate audible output(s), via a speaker, speaker jack, audio outputport, audio output device, earphones, and/or other similar devices.

In some embodiments, user interface 204 may include a display thatserves as a viewfinder for still camera and/or video camera functionssupported by computing device 200. Additionally, user interface 204 mayinclude one or more buttons, switches, knobs, and/or dials thatfacilitate the configuration and focusing of a camera function and thecapturing of images (e.g., capturing a picture). It may be possible thatsome or all of these buttons, switches, knobs, and/or dials areimplemented as functions on a presence-sensitive panel.

Processor 206 may comprise one or more general purpose processors—e.g.,microprocessors—and/or one or more special purpose processors—e.g.,digital signal processors (DSPs), graphics processing units (GPUs),floating point units (FPUs), network processors, or application-specificintegrated circuits (ASICs). In some instances, special purposeprocessors may be capable of image processing, image alignment, andmerging images, among other possibilities. Data storage 208 may includeone or more volatile and/or non-volatile storage components, such asmagnetic, optical, flash, or organic storage, and may be integrated inwhole or in part with processor 206. Data storage 208 may includeremovable and/or non-removable components.

Processor 206 may be capable of executing program instructions 218(e.g., compiled or non-compiled program logic and/or machine code)stored in data storage 208 to carry out the various functions describedherein. Therefore, data storage 208 may include a non-transitorycomputer-readable medium, having stored thereon program instructionsthat, upon execution by computing device 200, cause computing device 200to carry out any of the methods, processes, or functions disclosed inthis specification and/or the accompanying drawings. The execution ofprogram instructions 218 by processor 206 may result in processor 206using data 212.

By way of example, program instructions 218 may include an operatingsystem 222 (e.g., an operating system kernel, device driver(s), and/orother modules) and one or more application programs 220 (e.g., camerafunctions, address book, email, web browsing, social networking, and/orgaming applications) installed on computing device 200. Similarly, data212 may include operating system data 216 and application data 214.Operating system data 216 may be accessible primarily to operatingsystem 222, and application data 214 may be accessible primarily to oneor more of application programs 220. Application data 214 may bearranged in a file system that is visible to or hidden from a user ofcomputing device 200.

Application programs 220 may communicate with operating system 222through one or more application programming interfaces (APIs). TheseAPIs may facilitate, for instance, application programs 220 readingand/or writing application data 214, transmitting or receivinginformation via communication interface 202, receiving and/or displayinginformation on user interface 204, and so on.

In some vernaculars, application programs 220 may be referred to as“apps” for short. Additionally, application programs 220 may bedownloadable to computing device 200 through one or more onlineapplication stores or application markets. However, application programscan also be installed on computing device 200 in other ways, such as viaa web browser or through a physical interface (e.g., a USB port) oncomputing device 200.

Camera components 224 may include, but are not limited to, an aperture,shutter, recording surface (e.g., photographic film and/or an imagesensor), lens, and/or shutter button. Camera components 224 may becontrolled at least in part by software executed by processor 206.

Captured digital images may be represented as a one-dimensional,two-dimensional, or multi-dimensional array of pixels. Each pixel may berepresented by one or more values that may encode the respective pixel'scolor and/or brightness. For example, one possible encoding uses theYCbCr color model. In this color model, the Y channel may represent thebrightness of a pixel, and the Cb and Cr channels may represent the bluechrominance and red chrominance, respectively, of the pixel. Forinstance, each of these channels may take values from 0 to 255 (i.e.,the tonal range that a single 8-bit byte can offer). Thus, thebrightness of a pixel may be represented by a 0 or a value near zero ifthe pixel is black or close to black, and by a 255 or a value near 255if the pixel is white or close to white. However, the value of 255 is anon-limiting reference point, and some implementations may use differentmaximum values (e.g., 1023, 4095, etc.).

Nonetheless, the YCbCr color model is just one possible color model, andother color models such as a red-green-blue (RGB) color model or acyan-magenta-yellow-key (CMYK) may be employed with the embodimentsherein. Further, the pixels in an image may be represented in variousfile formats, including raw (uncompressed) formats, or compressedformats such as Joint Photographic Experts Group (JPEG), PortableNetwork Graphics (PNG), Graphics Interchange Format (GIF), and so on.

Some pixel encodings—including the YCbCr color model—use 8 bits torepresent the brightness of each pixel. Doing so is referred to as LDRimaging. As a result, only 256 levels of brightness may be supported.However, real-world scenes often exhibit a wider dynamic range ofbrightness than can be reasonably represented by LDR imaging. Forexample, a scene of an individual standing in a dark room in front of awindow may include both extremely bright regions and extremely darkregions. However, use of LDR imaging to capture an image of such a scenemay result in loss of detail in the bright region and/or the dark regionbased on the TET with which the image was captured.

A short TET may result in a reasonably accurate representation of thebright regions of a scene, but underexposure of the dark regions.Conversely, a long TET may result in a reasonably accuraterepresentation of the dark regions, but may overexpose the brightregions. In the example scene introduced above, if the TET is too long,the features in the room may appear properly-exposed, but the featuresoutside the window may appear whitewashed. But if the TET is too short,the features outside the window may appear normal but the features inthe room may appear darkened. Either of these outcomes is undesirable.For some scenes, there may not be a single TET that results in acaptured image representing the details in both bright regions and darkregions with acceptable detail.

Camera devices may support an auto-exposure (AE) mode in which, prior tooutput image capture, the camera determines the TET based on thebrightness of the scene. For example, the user may observe the scene inthe camera's viewfinder before triggering image capture. During thisperiod, the camera may make an initial estimate of the proper TET,capture a preview image with that TET, and then evaluate the pixels inthe captured image. Then, as one possible implementation, if a majority(or some other sufficient fraction) of the pixels in the preview imageare over-exposed, the camera may decrease the TET and capture anotherpreview image. If a majority (or some other sufficient fraction) of thepixels in this preview image are under-exposed, the camera may increasethe TET and capture yet another preview image.

For instance, if the majority of the pixels in the captured imageexhibit a brightness value above a high threshold level (e.g., 240), thecamera may decrease the TET. On the other hand, if a majority of thepixels exhibit a brightness level below a low threshold level (e.g.,96), the camera may increase the TET.

Alternatively or additionally, a target average pixel value for some orall of the scene's pixels may be determined. If the actual average pixelvalue is above the target average pixel value, the TET may be decreased,and if the actual average pixel value is below the target average pixelvalue, the TET may be increased. The target average pixel value can alsobe tuned differently depending on how much contrast there is in thescene. For example, in a low-contrast scene, the target average pixelvalue may be bright (e.g., 200). But in a high-contrast scene, thetarget average pixel value may be lower (e.g., 128).

This process may continue until the camera determines that an imageshould be captured and stored (e.g., the user activates the shutterbutton). During this process, if the characteristics of the scene arerelatively unchanging, the camera usually converges on an estimated“best” TET based on the brightness of the scene. In some embodiments,the image displayed on the camera's viewfinder may omit information fromone or more of the captured preview images or combine information fromtwo or more of the captured preview images.

In some cases, the camera might not treat all pixels equally whendetermining an “average” brightness of the scene. Using a techniquedescribed as “center-weighted averaging,” pixels near the middle of thescene may be considered to be more important. Thus, these pixels may beweighted more than pixels illustrating other areas of the scene.Alternatively, pixels in other locations of an image may be given moreweight. For instance, if the camera detects a human face (or some otherobject of interest) in a particular location other than the center ofthe image, the camera may give a higher weight to the associated pixels.

In this way, AE algorithms may seek to determine a TET that produces alarge number (e.g., the largest number) of properly-exposed pixels.However, given the range limitations of LDR imaging, even imagescaptured in AE mode may contain portions that are whitewashed ordarkened. Thus, as noted above, some scenes there may be no single“best” TET.

AE algorithms may differ from the description above. For instance, somemay be more complex, treating different colors differently, consideringthe spatial and/or structural components of a scene, and/or measuringcontrast between regions. The embodiments herein, however, may operatewith any AE algorithm now known or developed in the future.

High dynamic range (HDR) imaging has been proposed as a way ofcompensating for the deficiencies of LDR imaging. In a possibleimplementation, HDR imaging may involve a camera capturing multipleimages of a scene at various TETs, and then digitally processing thesecaptured images to make a single image that contains a reasonablerepresentation of the details in most or all regions of the scene,including those that are very bright and very dark. However, determiningTETs for capturing images can be problematic. In particular, difficultyin adjusting TETs for a particular scene has created limitations in HDRimaging. The methods and implementations described herein may providecomputational efficiency, robustness to artifacts, and/or enhanced imagequality.

In the following, the term “LDR image” may refer to an image capturedusing LDR imaging, and the term “LDR scene” may refer to a scene thathas been determined to be reasonably represented using LDR imaging.Similarly, the term “HDR image” may refer to an image captured using HDRimaging, and the term “HDR scene” may refer to a scene that has beendetermined to be reasonably represented using HDR imaging. Furthermore,the term “LDR imaging” may be used interchangeably with the term “LDRimage acquisition,” and the term “HDR imaging” may be usedinterchangeably with the term “HDR image acquisition.”

FIG. 3 depicts a flow chart, in accordance with an example embodiment.At a high level, flow chart 300 represents an imaging pipeline for adigital camera device 302. For instance, flow chart 300 may represent anumber of steps performed by digital camera device 302 to determinewhether to use LDR or HDR image acquisition for a particular scene,determine one or more TETs with which to capture images of the scene,and whether and/or how to combine the captured images into an outputimage that is a reasonably satisfactory representation of the scene. Inthis way, digital camera device 302 can dynamically adapt to thelighting pattern of the scene, whether that pattern is dark, bright, orsome combination of both. Digital camera device 302 may have the same orsimilar capabilities as digital camera device 100 in FIG. 1.

Flow chart 300 may represent a series of steps performed by digitalcamera device 302 when a shutter button is triggered. Alternatively oradditionally, flow chart 300 may represent steps that are continuouslyperformed when a viewfinder of digital camera device 302 is displaying arepresentation of a scene. Thus, in some embodiments, the features offlow chart 300 may be performed in a way that is not apparent to theuser. For instance, the user may trigger the shutter once, with theintention of capturing a single image. However, digital camera device302 may capture multiple images in each of first image capture 306 andsecond image capture 314, and provide an output image 318 that is acombination of one or more images captured during second image capture314.

It should be noted that not all steps depicted in FIG. 3 need beperformed by digital camera device 302. Some steps, such as imageprocessing 308 and combining images 316, for example, could be performedby a different device. For instance, representations of one or moreimages captured during first image captures 306 and second image capture314 could be transmitted from a capturing device to a remote computingdevice. The remote computing device could them perform image processing308 and combining images 316, possibly transmitting some or all of theresults thereof to the capturing device.

Additionally, training image database 304 may be included in digitalcamera device 302 or alternatively, training image database 304 may bepart of a separate device or system that may be accessed by digitalcamera device 302. In some embodiments, training image database 304 mayinclude representations of training images that can be used to helpdetermine the structure of a payload burst used in second image capture314.

In first image capture 306, a first group of images of a scene may becaptured using a “metering burst sweep.” In a metering burst sweep, eachimage in the group may be captured with a different TET. In someinstances, the metering burst sweep may capture consecutive imagesacross a range of TETs (e.g., 1-300 milliseconds, 0.1-500 milliseconds,or some other range). Using such ranges of TETs, the metering burstsweep may capture a series of images with TETs designed to cover thisrange according to a linear, logarithmic, and/or exponentialdistribution of TETs, among other possibilities.

As an example, FIG. 3 depicts first image capture 306 including threedigital images of a scene, each captured with a different TET. The threeimages exhibit diverse levels of brightness due to the different TETsused to capture the images. In other examples, more or fewer images maybe captured during first image capture 306. These captured images mayprovide parameters for digital camera device 302 to use when capturingsubsequent images of the scene.

The metering burst sweep can be used to determine the characteristics ofthe scene so that a subsequent payload burst structure for second imagecapture 314 can be selected. Therefore, in step 308, the images capturedat step 306 may be processed. Particularly, step 308 may include mergingone or more of the images captured at step 306 in a combined image 310.Step 308 may also include forming a histogram 312 from the mergedimages, and then using the histogram, and possibly some or all of theinformation in training image data 304, to classify the scene (e.g., asan LDR scene or an HDR scene), determine the structure of the payloadburst based on the classification of the scene, and determine the TETsto use when capturing images according to the payload burst. In someembodiments, the captured images, shown as a result of first imagecapture 306, may be downsampled prior to merging. Further, the histogrammay be an LDR histogram, HDR histogram, a log HDR histogram, or someother form of histogram.

An example of this process is illustrated in FIG. 4A. For sake ofsimplicity, assume that the metering burst sweep consist of two images,image 400 and image 402. In practice, anywhere from one to eight or moreimages may be captured in the metering burst sweep. Image 400 wascaptured with a TET of 10 milliseconds, while image 402 was capturedwith a TET of 20 milliseconds. Thus, the pixels of image 402 might beexpected to be about twice as bright as those of image 400. In somecases, the pixels in each of images 400 and 402 may be tonemapped.Therefore, the tonemapping process may be reverse for these images.After the tonemapping process is reversed, the pixels of image 402 maybe about twice as bright as those of image 400. Tonemapping and reversetonemapping are discussed in more detail below.

Images 400 and 402 may be downsampled to form images 404 and 406respectively. Downsampling can be implemented by, for example, dividinga given image into i×j pixel blocks (i and j may take on the same valueor different values), and replacing each of these blocks by a singlepixel. The value of this replacement pixel can be based on the values ofthe pixels in the respective i×j pixel block. For instance, the value ofthe replacement pixel may be determined by taking an average of thevalues of the pixels in the block, resulting in a “fuzzier,”lower-resolution, and smaller downsampled image. Thus, as one possibleexample, if a 1600×1200 pixel image is divided into 2×2 pixel blocks anddownsampled one level, the result is an 800×600 pixel image. If the1600×1200 pixel image is downsampled two levels (or if the 800×600 pixelimage is downsampled one more level), the result is a 400×300 pixelimage, and so on. Nonetheless, a tile can be downsampled in other ways.For example, a 4×4, an 8×8 pixel block or a 16×16 pixel block can bereplaced by a single pixel, and more than just one or two levels ofdownsampling can be performed.

In some embodiments, multiple levels of downsampling may be performedfor each image, thus creating a “pyramid” of downsampled images. Byusing images with multiple levels of downsampling, information regardingboth the distribution of light levels in the image and the spatialstructure of these light levels may be preserved.

Thus, for instance, FIG. 4A depicts image 400 having pixel block 400A inits top, left corner and pixel block 400B in its top right corner.Additionally, image 402 has pixel block 402A in its top, left corner andpixel block 402B in its top right corner. Each pixel block in each imageis downsampled to an individual respective pixel in images 404 and406—pixel 404A represents a downsampling of pixel block 400A, pixel 404Brepresents a downsampling of pixel block 400B, pixel 406A represents adownsampling of pixel block 402A, and pixel 406B represents adownsampling of pixel block 402B.

In some cases, an i×j pixel block and its associated downsampled pixelmay both be referred to as a “paxel.” Thus, pixel block 402A and pixel404A may both be referred to as a paxel.

For each location in the downsampled images (e.g., pixels 404A and 406Awould be considered to be in the same location), the pixel with thehighest value less than 255 may be selected. In some embodiments, thepixel value of each color channel in the pixel may be compared to 255.If all of these pixel values are below 255, then the pixel is acandidate for selection. Out of all of the candidate pixels, the onewith the highest pixel value may be selected. In other embodiments, athreshold different from 255 (e.g., 250, 245, or a value higher than255) may be used instead.

FIG. 4A illustrates an example of this process. In downsampled image404, pixel 404A may have red, green, and blue (R, G, B) values of 250,100, and 150, respectively. In downsampled image 406, pixel 406A mayhave (R, G, B) values of 255, 200, and 255, respectively. Since the (R,G, B) values of pixel 404A are all less than 255 but some of the (R, G,B) values of pixel 406A are at 255, pixel 404A is selected. Similarly,pixel 404B may have (R, G, B) values of 100, 50, and 50, respectively,while pixel 406B may have (R, G, B) values of 200, 100, and 100,respectively. Since the (R, G, B) values of pixel 406B are all less than255, but greater than the (R, G, B) values of pixel 404B, pixel 406B isselected. Other types of comparisons may be used also or instead of theprocess illustrated in FIG. 4A. For instance, the average of the (R, G,B) values of the respective pixels may be compared. Alternatively, theluma (Y) value of the pixel in the YCbCr color space may be used to testagainst the threshold.

Each selected pixel may be placed in its respective location in combinedimage 408. Thus, for instance, pixel 404A may be placed as pixel 408A,and pixel 406B may be placed as pixel 408B in combined image 408.Additionally, the pixels selected for combined image 408 may be reversetonemapped.

Tonemapping is a set of procedures that include mapping the pixel valuesaccording to a pre-determined function. Thus, for instance, some cameradevices map pixel values from a linear space (wherein an increase ordecrease of k units in pixel value represents a proportional increase ordecrease in brightness) to a non-linear space. Tonemapping may beautomatically performed for artistic purposes, such as brighteningmid-range pixel values. Regardless, in order to reverse tonemap theselected pixels back to linear space, the inverse of the tonemappingfunction may be applied to the respective pixel values.

Additionally, the pixel values may be divided by the respective TET withwhich the pixels were captured. Doing so may normalize the pixel valuesthat were captured using the various TETs to a particular range. Thus,for pixel 408A, the (R, G, B) values may be 25, 10, 15, while for pixel408B, the (R, G, B) values may be 10, 5, 5. On the other hand, for TETsunder 1 millisecond (e.g., TETs of 0.25 milliseconds, 0.5 milliseconds,and so on) dividing by the TET may increase the pixel values in combinedimage 408. In some cases, this may result in these pixel values beinggreater than 255, and therefore combined image 408 may be an HDRrepresentation of the scene.

Further, a histogram 410 may be created from combined image 408. Whilethere are many ways in which the histogram can be formed, someembodiments may include evaluating a function of the color channels ofeach pixel. This function may be a maximum or some other function, forinstance.

Moreover, the logarithm of this function's output may also be taken, andthe resulting value plotted on histogram 410. The distribution of lightlevels in the real world is logarithmic. Thus, by using a log scale,there is a roughly uniform coverage of that range. For histograms basedon linear light levels, more histogram bins might be used. Further, insome embodiments, a weight may be applied to the logarithm of thefunction's output before placing this output in histogram 410. Forinstance, a center-weighted average technique may be used to apply ahigher weight for pixels that are closer to the center of the capturedimage, and a lower weight might for pixels that are further from thecenter of the image.

Regardless, histogram 410 may represent the distribution of pixel valuesin combined image 408. The vertical axis of the histogram may indicatethe number of pixels of each pixel value and the horizontal axis mayrepresent the range of pixel values. The pixel values may be within therange 0-255, or some other range may be used. For instance, an HDRhistogram may include pixel values above 255. In some embodiments, anHDR histogram may represent 15-bit pixel values, i.e., from 0 to 32,767.Thus, the log values appearing in the HDR histogram may be in the rangeof 0 to log(32,767)=4.52.

As an example, histogram 410 plots most of the pixels on the extremeends of its horizontal axis. This indicated that most of histogram 410plots image 408 is a shade of black or white. However, since histogram410 also plots data points in the middle of the horizontal axis,combined image may contain pixels with mid-range brightness as well.

Still as part of image processing 308, histogram 410 may be compared toone or more histograms of images in training image database 304. Asdepicted in FIG. 4B, training image database 304 may contain a set ofhistograms of reference scenes, and/or associated parameters. Thehistograms of the reference scenes may be HDR histograms, as describedabove.

FIG. 4B shows training image database 304 containing an entry forhistogram 420, target pixel value(s) 422, target percentage of clippedpixels 424, and scene type 426. Histogram 420 may have been derived fromone or more images in the same or a similar fashion as described in thecontext of FIG. 4A, above. Thus, one or more images of a reference scenemay be captured using various TETs, and these images may be downsampledand combined into a single image from which histogram 420 is derived. Ingeneral, the reference scene from which histogram 420 is derived neednot be the same scene from which histogram 410 is derived.

In some embodiments, the percentage of pixels clipped (e.g., pixelshaving a pixel value of 255 or higher) in the downsampled, combinedsingle image may be stored in target percentage of clipped pixels 424.Clipping may occur when the TET used to capture the image was either toohigh or too low, and some very dark or very bright details of the scenewere lost in the image capture process. In some cases, only pixelsclipped at one end of the range may be included in the percentage ofpixels clipped. For instance, the percentage of pixels clipped mayinclude only pixels with a pixel value of 255 or higher, and the averagepixel value of unclipped pixels may be calculated over all other pixels(e.g., all pixels with a pixel value of 0-254).

Further, an indication of whether the scene was LDR or HDR may be storedin scene type 426. In some implementations, scene type 426 may take on avalue of zero when the scene type is LDR, and a value of one when thescene type is HDR. Alternatively, scene type 426 may be within a range,e.g., from 0 to 1. In this case, a value less than a threshold (e.g.,0.5) might indicate an LDR scene type, and any number greater than orequal to the threshold value might indicate an HDR scene type.

Target pixel value(s) 422 may be one or more pixel values that have beendetermined to be desirable for the associated scene. If the scene isLDR, target pixel value(s) 422 may include a single pixel value, or arange of pixel value(s). This single pixel value (or range) may havebeen selected so that images of the scene with an average pixel valuematching the target pixel value would be reasonably well-exposed. Thus,the target pixel value(s) 422 may be a target average pixel value.Additionally, the target pixel value(s) 422 may be LDR values (e.g.,from 0 to 255).

If the scene is HDR, target pixel value(s) 422 may include a shortexposure target pixel value (or range), a long exposure target pixelvalue (or range), and possibly a fallback exposure target pixel value(or range). These pixel values or ranges may have been selected so thatHDR images with the short exposure target pixel value for the shortexposures and long exposure target pixel value for the long exposuresmay be reasonable well exposed. The fallback target pixel value may beused if HDR imaging fails (e.g., as discussed below) and a single TET isused to capture the HDR scene.

In some cases, target pixel value(s) 422, target percentage of clippedpixels 424, and scene type 426 may be determined manually by examiningseveral synthetic exposures of the captured images and selecting thepixel value(s) and/or range(s) that appear most pleasing to the eye.Alternatively, target pixel value(s) 422, target percentage of clippedpixels 424, and scene type 426 may be determined algorithmically orautomatically.

Training image database 304 may also contain similar entries forhistogram 430, target pixel value(s) 432, target percentage of clippedpixels 434, and scene type 436, as well as histogram 440, target pixelvalue(s) 442, target percentage of clipped pixels 444, and scene type446. In some embodiments, one or more parameters, such as the targetpercentage of clipped pixels, may be omitted from training imagedatabase 304. Additionally, other parameters may be included in trainingimage database 304. Training image database may store as few as one orseveral tens, hundreds, or thousands of such entries, each entrypossibly relating to a different scene.

As discussed above in the context of an image captured during firstimage capture 306, multiple levels of downsampling may be performed foreach image in training image database 304, thus creating a “pyramid” ofdownsampled versions of each image. By using images with multiple levelsof downsampling, information regarding both the distribution of lightlevels in the image and the spatial structure of these light levels maybe preserved. Histograms for each of these downsampled images may beincluded in training image data 304.

The information in training image database 304 may be stored thereinduring calibration, testing, and/or other pre-shipment evaluationsbefore commercial operation, among other possibilities. Alternatively,the information may also be stored on various other devices and systemscapable of managing training image database 304. Regardless, theinformation may be substantially static in nature, though theinformation may be modified through firmware and/or software updates orother installations.

By comparing histogram 410 (and or a similarly-derived histograms basedon downsampled versions of images 400 and 402) with the histograms intraining image data 304, the dynamic range of the scene represented inimages 400 and 402 may be estimated. This estimated dynamic range may beused to determine, obtain, or otherwise select a payload burst structurefor second image capture 314.

For example, each histogram 410 may be compared to each histogram storedin training image data 304. A pair of histograms can be compared invarious ways, any of which may be used. In some embodiments, the earthmover's distance (EMD) between the pair of histograms may be calculated.The EMD is 0 when the histograms are identical, and increases with thedifferences between the histograms. Thus, a lower EMD indicates a goodmatch between the two histograms, and a higher EMD indicates a poormatch between the two histograms.

A weight may be derived from the EMD for a particular pair ofhistograms. For example, the weight may be inversely proportional to itsassociated EMD value. In some situations, the weight w may be derivedas:

$w = \frac{1}{{EMD}^{n}}$

where EMD is the EMD value and n may be in the range of 1 to 10.However, other values of n may be used. Thus, in some embodiments, theweight may take on a value between 0 and 1. In the case that EMD is 0,and appropriately large weight (e.g., 1) may be chosen. Alternatively, avery small value (e.g., 0.001) may be added to the denominator to avoiddividing by 0. Regardless, a good match between the two histograms mayresult in a high weight, and a poor match between the two histograms mayresult in a low weight.

For each pair of histograms (where a first histogram is histogram 410and a second histogram is from an entry in training image database 304),the weight w may be applied to the respective scene type associated withthe second histogram's entry in training image database 304. The resultmay be averaged over the pairs of histograms to determine an “HDR-ness”of the scene. For instance, if the resulting “HDR-ness” is at or above0.5 on a scale of 0 to 1, then the scene may be designated for HDRprocessing, but if the resulting “HDR-ness” is below 0.5 on the samescale, then the scene may be designated for LDR processing.

It should be understood that the embodiment described above is just onepossible way of determining the brightness and “HDR-ness” of a scene.Other techniques could be used instead, and some of these othertechniques may be based on comparing the parameters (e.g., percentage ofclipped pixels and average value of unclipped pixels) of the imagescaptured during first image capture 306 with the same or similarparameters of the images represented in training image database 304.Further techniques may include comparing the respective downsampledimage pyramids of the first and second histograms.

In order to determine the TETs for the payload burst, the followingexample procedure may be used. However, other procedures may be usedinstead.

For each pair of histograms (again, where the first histogram ishistogram 410 and a second histogram is from an entry in training imagedatabase 304), the weight w may be applied to the respective targetpixel value associated with the second histogram's entry in trainingimage database 304. The result may be averaged to determine a targetaverage pixel value for the scene represented by histogram 410. If thescene type is HDR, two or more target average pixel values may bedetermined.

For each target average pixel value, interval halving may be used tosearch for a particular TET value such that if the scene were to becaptured with the particular TET value, the resulting image would havethe target average pixel value or about the target average pixel value.One possible method for determining a particular TET value based on atarget average pixel value is shown in the pseudocode below.

TABLE 1 1. lo_tet = 0.125 ms 2. hi_tet = 66.6 * 8 * 4 ms 3. for (int i =1; i < t; i ++) { 4.   mid_tet = (lo_tet + hi_tet) / 2 5.  pixel_value_at_mid_tet = <synthetically expose the HDR    Image at‘mid_tet’, tonemap it, and extract the    average pixel value of theresulting LDR image.> 6.   if (pixel_value_at_mid_tet >target_average_pixel_value) { 7.     hi_tet = mid_tet 8.     } else { 9.    lo_tet = mid_tet 10.     } 11. }

At lines 1 and 2 of Table 1, initial low and high TET values (lo_tet andhi_tet, respectively) are defined. These values may be chosen at or nearthe extreme ends of the range in which the ultimate TET value (mid_tet)is expected to fall. In some embodiments, a broader range or a narrowerrange may be used.

Lines 3-11 depict a loop that may be iterated t times. The value of tmay be selected so that mid_tet converges after t iterations of theloop. In some embodiments t may be as low as 2 or 3, but in otherembodiments, t may be 5, 10, 20, 50, 100, or some other value.Nonetheless, at line 4, mid_tet is set to be the average (midpoint) oflo_tet and hi_tet.

At line 5, the pixel value of the image at mid_tet is determined. Onepossible way of making this determination is to synthetically expose theHDR image as if the image were captured using a TET of mid_tet. Theresulting image (which may be an HDR image) may be tonemapped, and theaverage pixel value of the tonemapped image (pixel_value_at_mid_tet,which may be an LDR value) may be determined.

Synthetic exposure is one way of obtaining an LDR image from an HDRimage. Suppose that an HDR image was captured using a TET of T. This HDRimage can be synthetically exposed to a TET of p times T by multiplyingthe pixel values of each pixel in the HDR image by p (p may be greaterthan or less than one). In the resulting image, all pixel values above255 are “clipped” to 255. This process simulates the appearance of thescene as if it were captured using a TET of p times T with LDR imaging.Alternatively, a non-log HDR histogram of an HDR image (with or withoutcenter-weighted averaging applied) may be used. After this step,whatever processing would normally be applied to the linear image (suchas tonemapping) may be emulated, in order to produce a synthetic LDRimage. The average value in that image (applying center-weightedaveraging if desired) may be taken and compared to the target pixelvalue.

At lines 6-11, if this resulting average pixel value is greater than thetarget average pixel value, then mid_tet is too high, and hi_tet is setto be mid_tet in order to reduce mid_tet in the next iteration. On theother hand, if the resulting average pixel value is less than or equalto the target average pixel value, then the mid_tet is too low, andlo_tet is set to be mid_tet in order to increase mid_tet in the nextiteration.

The process illustrated by Table 1 may be repeated for each TET valuethat may be used in the payload burst structure. Thus, if the scene isdetermined to be an LDR scene, the process of Table 1 may be carried outfor one TET. However, if the scene is determined to be an HDR scene, theprocess of Table 1 may be carried out for two or more TETs (e.g., theshort TET, the long TET, and/or the fallback TET), all three of whichmay have different target average LDR pixel values.

In step 314, the second group of images may be captured. The number ofimages captured and the arrangement of TETs used to capture these imagesmay be referred to as a “payload burst.” For example, in FIG. 3 secondimage capture 314 includes three images of a scene, each captured with aTET identified in step 308. It should be understood that the TETsidentified in step 308 may be the same or different than the TETs usedto capture images in step 306. Additionally, it is possible that allthree images in second image capture 314 are captured with the same orsimilar TETs.

In step 316, images from the second group of images may be combined.Combining images may include aligning two or more of the images. In someinstances, images may be aligned globally (i.e., aligning whole imagesas opposed to portions of images), locally (i.e., aligning portions ofimages), or possibly both globally and locally. Further, combining twoor more images may also include merging them to form an output image318. This merging may be carried out in accordance with any image fusiontechnique now known or developed in the future.

Merging the images in the second group of images may result in outputimage 318 being sharper and/or better-exposed than any of the individualimages in the second group. For instance, if some of the images insecond image capture 314 are captured with the same or similar TETs,these images may be merged to reduce noise in one or more sections ofthe images. Alternatively or additionally, if the images in second imagecapture 314 are captured with two or more different TETs, at least someimages with different exposure times may be merged according to HDRprocedures. Regardless, the output image may be stored on acomputer-readable medium and/or displayed on an output medium such asthe multi-element display 106 of FIG. 1.

In some embodiments, the arrangements of various possible payload burststructures may be determined based on the TETs determined in step 308,as well as an understanding of combining images step 316. While numerousarrangements of payload burst structures may be possible, three examplesare described herein.

TABLE 2 Scene Type Payload Burst Structure LDR T T T T HDR L S L L S L LS L L HDR (with fallback) L S L L S L L F F F

Table 2 illustrates these examples. In the first example, the scene typeis LDR. In this example, the payload burst structure includes fourimages captured sequentially, and may be referred to as an “LDR burststructure.” Each “T” in the Payload Burst Structure column of Table 2may represent a captured image. Each of these images may be capturedusing the same or a similar TET that was determined in step 308. In someembodiments, fewer or more images may be captured in an LDR payloadburst. For example, as few as one, or as many as ten or more images maybe included.

Regardless of the number of images captured, some of these images may bealigned and combined in step 316. For instance, if m images are capturedin the payload burst, the sharpest one of these images may be selectedas a “primary image,” and the remaining m−1 images may be considered“secondary images.” In some implementations, the sharpness of an imagemay be measured by the image's resolution and/or boundaries betweenzones of different tones and/or colors in the image. Alternatively oradditionally, other sharpness measurements may be used.

Further, zero or more of the m−1 secondary images may then be alignedand merged with the sharpest image. For instance, alignment may beattempted between each of the secondary images and the sharpest image,respectively. If the alignment fails for parts of a respective secondaryimage, those parts may be discarded, and not combined with the primaryimage. In this way, the sharpest image may be denoised with informationfrom some or all of the secondary images.

In the second example, the scene type is HDR. In this example, thepayload burst structure includes ten images captured according to apattern of long and short TETs, and may be referred to as an “HDR burststructure.” In the Payload Burst Structure column of Table 2, each “L”may represent an image captured with the long TET, each “S” mayrepresent an image captured with the short TET. Thus, the pattern of “LS L L S L L S L L” may indicate that the first image of the payloadburst is captured using the long TET, the second image is captured usingthe short TET, the third and fourth images are captured using the longTET, the fifth image is captured using the short TET, the sixth andseventh images are captured using the long TET, the eighth image iscaptured using the short TET, the ninth image is captured using the longTET, and the tenth image is captured using the long TET.

The long and short TETs may be determined based on the results of imageprocessing 308. Thus, the long and short TETs may be selected so thatthe resulting images captured with these TETs can be combined using HDRprocedures. The long TET may be used to capture the details in darksections of the scene, while the short TET may be used to capture thedetails in bright sections of the scene.

Examples of short TET values may include TETs of 1 millisecond, 2milliseconds, and/or 8 milliseconds, while examples of long TET valuesmay include TETs of 20 milliseconds, 40 milliseconds, and/or 80milliseconds. However, short and long TETs may take on different values.

Despite the payload burst in the second example having a particularstructure in Table 1, other structures may be used. For example, payloadburst structures of “L S L S L S L S L S” or “L L S L L S L L S L” couldpotentially provide suitable patterns of long and short TETs. Further,some payload burst structures may include medium TETs (denoted by an“M”). Thus, additional example payload bursts may include “S M L L L S ML L L” or “S M L S M L S M L L” structures.

In some embodiments, a payload burst structure may include more or fewerthan ten images. Generally speaking, the determining the length of thepayload burst structure involves a tradeoff. On one hand, a long payloadburst (i.e., a payload burst with a large number of image captures) isdesirable because the likelihood of one or more of the captured imagebeing well-exposed and sharp is increased. On the other hand, if thepayload burst is too long, the likelihood of ghosting due to movement inthe scene is also increased. Additionally, darker scenes may benefitfrom images captured using a longer TET, so that more light can reachthe recording surface. Therefore, the payload burst structure may bebased, possibly in part, on these considerations.

In the third example, the scene type is also HDR. However, in thisexample, the associated payload burst structure (which also may bereferred to as an HDR burst structure) includes seven images capturedaccording to a pattern of long and short TETs, followed by threefallback TETs. Each “F” may represent an image captured with thefallback TET, and the fallback TET may take on a value different fromboth the long and short TETs.

Regardless of the type of payload burst structure, the images of an HDRscene may be aligned and combined. Images captured using the short TETmay be referred to as “short images” for convenience, and imagescaptured with the long TET may be referred to as “long images” forconvenience.

In some embodiments, the sharpest short image may be selected, from theshort images, as the primary short image. Zero or more of the remainingsecondary short images may then be aligned and merged with the primaryshort image. For instance, alignment may be attempted between each ofthe secondary short images and the primary short image, respectively. Ifthe alignment fails for parts of the respective secondary short image,those parts may be discarded, and not combined with the primary shortimage. In this way, the sharpest short image may be denoised withinformation from some of the secondary short images.

The same or a similar process may be undertaken for the long images. Forexample, the sharpest long image may be selected, from the long images,as the primary long image. Zero or more of the remaining secondary longimages may then be aligned and merged with the primary long image.Alignment may be attempted between each of the secondary long images andthe primary long image, respectively. If the alignment fails for partsof the respective secondary long image, those parts may be discarded,and not combined with the primary long image.

The resulting combined short image (e.g., the sharpest short imagepossibly denoised by information from zero or more secondary shortimages) and the resulting combined long image (e.g., the sharpest longimage possibly denoised by information from zero or more secondary longimages) may then be aligned. If the alignment succeeds, these two images(e.g., both LDR images) may be combined according to HDR procedures. Forinstance, they may be combined into an HDR image, and the HDR image maythen be tonemapped so that its brightness falls within a rangecommensurate with the display abilities of convention video outputdevices (e.g., pixel values between 0 and 255, inclusive). The resultingtonemapped HDR image may be designated as output image 318. In someembodiments, if the signal-to-noise ratio of part or all of output image318 is still lower than a threshold value, a de-noising procedure may beapplied to further reduce noise. Additionally, output image 318 may alsobe sharpened, possibly after applying the de-noising procedure. Ingeneral, various types of HDR fusion algorithms, such as Exposure Fusionor Local Laplacian Filters, may be used to merge short and long images.If medium TETs are used in the payload burst structure, these HDR fusionalgorithms may be applied to one or more medium images as well.

If the alignment fails between the combined short image and the combinedlong image, then the HDR processing fails. However, if fallback imageswere captured, one or more of the fallback images may be used to formoutput image 318. For instance, the sharpest fallback image may beselected. Zero or more of the remaining secondary fallback images may bealigned and combined with the sharpest fallback image carried out in asimilar fashion as the processes described above for the short and longimages. For payload burst structures without fallback images in whichalignment fails, the combined long or short image may be used to formoutput image 318.

Example payload burst structures are further illustrated in FIG. 5.Payload burst structure 500 may represent a payload burst for an LDRscene, payload burst structure 502 may represent a payload burst for anHDR scene, and payload burst structure 504 may represent a payload burstfor an HDR scene with fallback. Each burst structure shows theapproximate duration of the TET for each image capture. For example, forburst structures 502 and 504, short TETs are of a lesser duration thanlong TETs, and the duration of fallback TETs are between that of shortand long TETs.

In FIG. 5, the vertical dotted lines depict epochs in time at which animage capture can begin. Some electronic image sensors operate at aparticular frequency, such as 30 hertz. A sensor operating at thisfrequency can capture 30 images per second, or approximately one imageevery 33.33 milliseconds. Thus, the duration between the epochs in FIG.5 may be 33.33 milliseconds. Nonetheless, for low-light scenes, theexposure time may be longer than 33.33 milliseconds, and the operatingfrequency of the image sensor may be adjusted downward accordingly. Inthis example, a TET longer than 30 milliseconds can be achieved byincreasing the gain component of the TET.

For some sensors, image capture may be activated only at the end of suchan epoch. Thus, as shown in FIG. 5, each image capture ends at the edgeof a respective epoch. However, due to their varying TETs, some imagecaptures may begin at different times. Alternatively, for some imagesensors, image capture may be activated only at the beginning of anepoch.

It should be understood that various other techniques, procedures,and/or algorithms may be used determine a payload burst structure. Thus,the discussion above accompanying Table 2 and FIG. 5 above merelyprovides some possible payload burst structures. Other techniques may beused without departing from the scope of the embodiments herein.

For example, in some embodiments, the average pixel value of thetonemapped image (e.g., step 5 in Table 1) may be calculated usingeither an RMS or SMR method, based on the length of the TET. The RMSaverage of n values may be calculated as:

${RMS} = \sqrt{\frac{x_{1}^{2} + x_{2}^{2} + \ldots + x_{n}^{2}}{n}}$

The SMR average of n values may be calculated as:

${SMR} = ( \frac{\sqrt{x_{1}} + \sqrt{x_{2}} + \ldots + \sqrt{x_{n}}}{n} )^{2}$

For shorter TETs, e.g., the short TETs and/or fallback TETs of an HDRpayload burst, or possibly the TETs of an LDR burst, it may be desirableto attempt to increase or maximize the brightness of the pixels withoutclipping them. Thus, taking the RMS average of the pixel values putsmore weight on the brighter pixels. For longer, TETs, e.g., the longTETs of an HDR payload burst, it may be desirable to emphasize thedarker parts of the scene. Thus, taking the SMR average of the pixelvalues puts more weight on the darker pixels.

Another variation is illustrated in the context of burst structure 502of FIG. 5. This variation involves selecting the sharpest short image.For sake of argument, assume that the sharpest long image is long image508. Then, instead of selecting the sharpest short image as the primaryshort image, the short image that was captured closest in time to thesharpest long image may be selected as the primary short image. Thiscould be, for instance, short image 510, which immediately precedes longimage 508. Then, the remaining secondary short images may be alignedwith and/or combined into (as the alignments permit) the primary shortimage. Alternatively, if the image sensor captures images at thebeginning of the epoch, a short image following (perhaps immediatelyfollowing) the sharpest long image may be selected as the primary shortimage. As an alternative, a temporally-adjacent short image/long imagepair may be selected so that, together, these images maximize acombined-sharpness metric.

In some embodiments, the payload burst structure may be based on whetherthe image sensor captures images with an exposure time of less than thereadout time of the images sensor's pixels (referred to herein assub-readout exposure times) at the beginning or end of an image captureepoch. If the image sensor captures sub-readout exposure times at theend of the image capture epoch, the payload burst structure may includeone or more two-TET subsequences of a short TET immediately followed bya long TET. If the image sensor captures sub-readout exposure times atthe beginning of the image capture epoch, the payload burst structuremay include one or more two-TET subsequences of a long TET immediatelyfollowed by a short TET.

One possible advantage of selecting the primary long image in thisfashion is to reduce motion blur, or “ghosting,” effects. For example,if a scene contains motion, merging multiple images captured from thescene may result in the motion appearing in the merged image as a blur.In general, the greater the difference in time between when the imagesare captured, the greater this blurring. By selecting primary long andshort images that are close to one another in time, the ghosting may bereduced

It should be noted that the steps illustrated by flow chart 300 may becarried out by various types of cameras and/or computing devices, suchas those exemplified by digital camera device 302 and/or computingdevice 200. Further, it may be possible to distribute aspects of someindividual steps between multiple cameras and/or computing devices. Forexample, first image capture 306 and second image capture 314 may occuron digital camera device 302. Further, image processing 308 andcombining images 318 may occur on a different computing device. Othercombinations of distributing individual steps may also exist.

FIG. 6 depicts a flow chart, in accordance with an example embodiment.At step 600, a first plurality of images of a scene may be captured,perhaps by an image sensor. Each image of the first plurality of imagesmay be captured with a different total exposure time (TET).

At step 602, based at least on the first plurality of images, a TETsequence for capturing images of the scene may be determined.Determining the TET sequence may involve determining a scene histogrambased on at least one of the images in the first plurality of images ofthe scene. The scene histogram may be based on downsampling andcombining the images in the first plurality of images of the scene.

In some embodiments, additional histograms may be stored in a trainingimage database. These histograms may be based on at least two imagescaptured of respective scenes. The histograms may be associated withrespective dynamic range parameters, where the respective dynamic rangeparameters indicate whether the respective scenes are LDR or HDR.Determining the TET sequence may further involve comparing the scenehistogram to at least one histogram in the training image database, andbased on an outcome of the comparison, determining a dynamic rangeparameter for the scene from the respective dynamic range parameters,where the TET sequence is further based on the dynamic range parameterfor the scene.

If the dynamic range parameter for the scene indicates that the scene isLDR, then determining the TET sequence may involve selecting a singleTET value to use in the TET sequence. If the dynamic range parameter forthe scene indicates that the scene is HDR, then determining the TETsequence may involve selecting a short TET value and a long TET value touse in the TET sequence. If the scene is HDR, determining the TETsequence may also involve selecting a fallback TET value to use in theTET sequence.

In some embodiments, the histograms in the training database may also beassociated with respective target pixel values. In these embodiments,determining the TET sequence may further involve determining one or moretarget pixel values for the scene based on the respective target pixelvalues in the training database, selecting one or more TET values to usein the TET sequence based on the one or more determined target pixelvalues.

At step 604, a second plurality of images of the scene may be capturedby the image sensor. The images in the second plurality of images may becaptured using the TET sequence. At step 606, based at least on thesecond plurality of images, an output image of the scene may beconstructed.

Determining the TET sequence may involve determining that the scene isan LDR scene and defining a common value for TETs in the TET sequence.Constructing the output image of the scene may involve aligning andcombining one or more of the images in the second plurality of imagescaptured using the common value.

Alternatively or additionally, determining the TET sequence may involvedetermining that the scene is an HDR scene and defining a short TETvalue and a long TET value. Constructing the output image of the scenemay involve aligning and combining (i) one or more of the images in thesecond plurality of images captured with the short TET value, and (ii)one or more of the images in the second plurality of images capturedwith the long TET value.

Alternatively or additionally, determining the TET sequence may involvedetermining that the scene is an HDR scene and defining a short TETvalue, a long TET value, and a fallback TET value. Constructing theoutput image of the scene may involve attempting to align (i) one ormore of the images in the second plurality of images captured with theshort TET value, with (ii) one or more of the images in the secondplurality of images captured with the long TET value. Constructing theoutput image of the scene may further involve determining that alignmentof (i) the one or more of the images in the second plurality of imagescaptured with the short TET value, and (ii) the one or more of theimages in the second plurality of images captured with the long TETvalue has failed. Constructing the output image of the scene may alsoinvolve, in response to determining that the alignment has failed,aligning and combining one or more of the images in the second pluralityof images captured with the fallback TET value to form the output image.

FIG. 7 depicts another flow chart, in accordance with an exampleembodiment. At step 700, a first plurality of images of a scene may becaptured by an image sensor. Each image of the first plurality of imagesmay be captured using a different total exposure time (TET). At step702, based at least on the first plurality of images, a long TET, ashort TET, and a TET sequence that includes the long TET and the shortTET may be determined.

At step 704, a second plurality of images of the scene may be capturedby the image sensor, wherein the images in the second plurality ofimages are captured sequentially in an image sequence using a sequenceof TETs corresponding to the TET sequence. The image sequence mayinclude a three-image sub-sequence of a first long-TET image capturedusing the long TET, followed by a short-TET image captured using theshort TET, followed by a second long-TET image captured using the longTET. Alternatively, the image sequence may include a three-imagesub-sequence of a first long-TET image captured using the long TET,followed by a second long-TET image captured using the long TET,followed by a short-TET image captured using the short TET. The patternin the subsequence may be chosen based on characteristics of the cameradevice (e.g., whether a rolling shutter is used and/or other imagesensor characteristics) and/or characteristics of the scene (e.g.,average pixel value or some other metric). Regardless, the sub-sequencemay repeat one or more times through the payload burst structure.

At step 706, based on one or more images in the image sequence, anoutput image may be constructed. In some embodiments, the image sequencemay include a two-image sub-sequence of a primary short-TET imagefollowed by a primary long-TET image. The image sequence may alsoinclude one or more secondary short-TET images and one or more secondarylong-TET images. The primary short-TET image and the secondary short-TETimages may be captured using the short TET, and the primary long-TETimage and the secondary long-TET images may be captured using the longTET. Constructing the output image may involve forming a combinedshort-TET image, where the combined short-TET image includes the primaryshort-TET image and at least part of the one or more secondary short-TETimages, forming a combined long-TET image, where the combined long-TETimage includes the primary long-TET image and at least part of the oneor more secondary long-TET images, and forming the output image, wherethe output image includes at least part of the combined short-TET imageand at least part of the combined long-TET image.

Alternatively or additionally, the image sequence may include a long-TETimage captured using the long TET. Constructing the output image mayinvolve determining that the long-TET image is a sharpest image of allimages in the image sequence captured using the long TET. Based on thelong-TET image being the sharpest image of all images in the imagesequence captured using the long TET, selecting the long-TET image as aprimary long-TET image, and selecting, as a primary short-TET image, animage captured using the short TET that is adjacent to the primarylong-TET image in the image sequence. Constructing the output image mayfurther involve forming a combined short-TET image, where the combinedshort-TET image includes the primary short-TET image and at least partof one or more images from the image sequence that were captured usingthe short TET, forming a combined long-TET image, where the combinedlong-TET image includes the primary long-TET image and at least part ofone or more images from the image sequence that were captured using thelong TET, and forming the output image, where the output image includesat least part of the combined short-TET image and at least part of thecombined long-TET image.

In some examples, the primary short-TET image may immediately precedethe primary long-TET image in the image sequence. The short-TET imagethat immediately precedes the primary long-TET image in the imagesequence may be selected as the primary short-TET image based on imagecapture ending at the edge of respective image capture epochs.

In other examples, the primary short-TET image may immediately followthe primary long-TET image in the image sequence. The short-TET imagethat immediately follows the primary long-TET image in the imagesequence may be selected as the primary short-TET image based on imagecapture beginning at the edge of respective image capture epochs.

The steps depicted in FIGS. 6 and 7 may be carried out by a cameradevice, such as digital camera device 100, a computing device, such ascomputing device 200, and/or by two or more distinct devices. Forinstance, in some embodiments, the image capture steps may be performedby an image sensor and the remaining steps may be performed by aseparate computing device. Other arrangements are possible. Further, theflow charts depicted in FIGS. 6 and/or 7 may be modified according tothe variations disclosed in this specification and/or the accompanyingdrawings.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context indicates otherwise. The illustrativeembodiments described in the detailed description, figures, and claimsare not meant to be limiting. Other embodiments can be utilized, andother changes can be made, without departing from the spirit or scope ofthe subject matter presented herein. It will be readily understood thatthe aspects of the present disclosure, as generally described herein,and illustrated in the figures, can be arranged, substituted, combined,separated, and designed in a wide variety of different configurations,all of which are explicitly contemplated herein.

With respect to any or all of the message flow diagrams, scenarios, andflow charts in the figures and as discussed herein, each step, blockand/or communication may represent a processing of information and/or atransmission of information in accordance with example embodiments.Alternative embodiments are included within the scope of these exampleembodiments. In these alternative embodiments, for example, functionsdescribed as steps, blocks, transmissions, communications, requests,responses, and/or messages may be executed out of order from that shownor discussed, including in substantially concurrent or in reverse order,depending on the functionality involved. Further, more or fewer steps,blocks and/or functions may be used with any of the message flowdiagrams, scenarios, and flow charts discussed herein, and these messageflow diagrams, scenarios, and flow charts may be combined with oneanother, in part or in whole.

A step or block that represents a processing of information maycorrespond to circuitry that can be configured to perform the specificlogical functions of a herein-described method or technique.Alternatively or additionally, a step or block that represents aprocessing of information may correspond to a module, a segment, or aportion of program code (including related data). The program code mayinclude one or more instructions executable by a processor forimplementing specific logical functions or actions in the method ortechnique. The program code and/or related data may be stored on anytype of computer-readable medium, such as a storage device, including adisk drive, a hard drive, or other storage media.

The computer-readable medium may also include non-transitorycomputer-readable media such as computer-readable media that stores datafor short periods of time like register memory, processor cache, and/orrandom access memory (RAM). The computer-readable media may also includenon-transitory computer-readable media that stores program code and/ordata for longer periods of time, such as secondary or persistent longterm storage, like read only memory (ROM), optical or magnetic disks,and/or compact-disc read only memory (CD-ROM), for example. Thecomputer-readable media may also be any other volatile or non-volatilestorage systems. A computer-readable medium may be considered acomputer-readable storage medium, for example, or a tangible storagedevice.

Moreover, a step or block that represents one or more informationtransmissions may correspond to information transmissions betweensoftware and/or hardware modules in the same physical device. However,other information transmissions may be between software modules and/orhardware modules in different physical devices.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

What is claimed is:
 1. A method comprising: capturing, by an imagesensor, a first plurality of images of a scene, wherein each image ofthe first plurality of images is captured with a different totalexposure time (TET); based at least on the first plurality of images,determining a TET sequence for capturing images of the scene, whereindetermining the TET sequence comprises determining that the scene is ahigh dynamic range (HDR) scene and defining a short TET value, a longTET value, and a fallback TET value; capturing, by the image sensor, asecond plurality of images of the scene, wherein images in the secondplurality of images are captured using the TET sequence; and based atleast on the second plurality of images, constructing an output image ofthe scene, wherein constructing the output image of the scene comprises(a) attempting to align (i) one or more of the images in the secondplurality of images captured with the short TET value, with (ii) one ormore of the images in the second plurality of images captured with thelong TET value, (b) determining that alignment of (i) the one or more ofthe images in the second plurality of images captured with the short TETvalue, and (ii) the one or more of the images in the second plurality ofimages captured with the long TET value has failed, and (c) in responseto determining that the alignment has failed, aligning and combining oneor more of the images in the second plurality of images captured withthe fallback TET value to form the output image.
 2. The method of claim1, wherein determining the TET sequence comprises determining a scenehistogram based on at least one of the images in the first plurality ofimages of the scene.
 3. The method of claim 2, wherein the scenehistogram is based on downsampling and combining the images in the firstplurality of images of the scene.
 4. The method of claim 2, whereinhistograms are stored in a training database and are based on at leasttwo images captured of respective scenes, wherein the histograms in thetraining database are associated with respective dynamic rangeparameters, wherein the respective dynamic range parameters indicatewhether the respective scenes exhibit low dynamic range (LDR) or HDR,and wherein determining the TET sequence comprises: comparing the scenehistogram to at least one histogram in the training database; and basedon an outcome of the comparison, determining a dynamic range parameterfor the scene from the respective dynamic range parameters, whereindetermining the TET sequence is further based on the dynamic rangeparameter for the scene.
 5. The method of claim 4, wherein thehistograms in the training database are also associated with respectivetarget pixel values, wherein determining the TET sequence furthercomprises: based on the respective target pixel values in the trainingdatabase, determining one or more target pixel values for the scene; andbased on the one or more determined target pixel values, selecting oneor more TET values to use in the TET sequence.
 6. The method of claim 5,wherein the selected one or more TET values include the short TET value,the long TET value, and the fallback TET value.
 7. The method of claim1, wherein the fallback TET value is between the short TET value and thelong TET value.
 8. An article of manufacture including a non-transitorycomputer-readable medium, having stored thereon program instructionsthat, upon execution by a computing device, cause the computing deviceto perform operations comprising: capturing, by an image sensor, a firstplurality of images of a scene, wherein each image of the firstplurality of images is captured with a different total exposure time(TET); based at least on the first plurality of images, determining aTET sequence for capturing images of the scene, wherein determining theTET sequence comprises determining that the scene is a high dynamicrange (HDR) scene and defining a short TET value, a long TET value, anda fallback TET value; capturing, by the image sensor, a second pluralityof images of the scene, wherein images in the second plurality of imagesare captured using the TET sequence; and based at least on the secondplurality of images, constructing an output image of the scene, whereinconstructing the output image of the scene comprises (a) attempting toalign (i) one or more of the images in the second plurality of imagescaptured with the short TET value, with (ii) one or more of the imagesin the second plurality of images captured with the long TET value, (b)determining that alignment of (i) the one or more of the images in thesecond plurality of images captured with the short TET value, and (ii)the one or more of the images in the second plurality of images capturedwith the long TET value has failed, and (c) in response to determiningthat the alignment has failed, aligning and combining one or more of theimages in the second plurality of images captured with the fallback TETvalue to form the output image.
 9. The article of manufacture of claim8, wherein determining the TET sequence comprises determining a scenehistogram based on at least one of the images in the first plurality ofimages of the scene.
 10. The article of manufacture of claim 9, whereinthe scene histogram is based on downsampling and combining the images inthe first plurality of images of the scene.
 11. The article ofmanufacture of claim 9, wherein histograms are stored in a trainingdatabase and are based on at least two images captured of respectivescenes, wherein the histograms in the training database are associatedwith respective dynamic range parameters, wherein the respective dynamicrange parameters indicate whether the respective scenes exhibit lowdynamic range (LDR) or HDR, and wherein determining the TET sequencecomprises: comparing the scene histogram to at least one histogram inthe training database; and based on an outcome of the comparison,determining a dynamic range parameter for the scene from the respectivedynamic range parameters, wherein determining the TET sequence isfurther based on the dynamic range parameter for the scene.
 12. Thearticle of manufacture of claim 11, wherein the histograms in thetraining database are also associated with respective target pixelvalues, wherein determining the TET sequence further comprises: based onthe respective target pixel values in the training database, determiningone or more target pixel values for the scene; and based on the one ormore determined target pixel values, selecting one or more TET values touse in the TET sequence.
 13. The article of manufacture of claim 12,wherein the selected one or more TET values include the short TET value,the long TET value, and the fallback TET value.
 14. The article ofmanufacture of claim 8, wherein the fallback TET value is between theshort TET value and the long TET value.
 15. A computing devicecomprising: at least one processor; an image sensor; data storage; andprogram instructions, stored in the data storage, that upon execution bythe at least one processor cause the computing device to performoperations including: capturing, by the image sensor, a first pluralityof images of a scene, wherein each image of the first plurality ofimages is captured with a different total exposure time (TET); based atleast on the first plurality of images, determining a TET sequence forcapturing images of the scene, wherein determining the TET sequencecomprises determining that the scene is a high dynamic range (HDR) sceneand defining a short TET value, a long TET value, and a fallback TETvalue; capturing, by the image sensor, a second plurality of images ofthe scene, wherein images in the second plurality of images are capturedusing the TET sequence; and based at least on the second plurality ofimages, constructing an output image of the scene, wherein constructingthe output image of the scene comprises (a) attempting to align (i) oneor more of the images in the second plurality of images captured withthe short TET value, with (ii) one or more of the images in the secondplurality of images captured with the long TET value, (b) determiningthat alignment of (i) the one or more of the images in the secondplurality of images captured with the short TET value, and (ii) the oneor more of the images in the second plurality of images captured withthe long TET value has failed, and (c) in response to determining thatthe alignment has failed, aligning and combining one or more of theimages in the second plurality of images captured with the fallback TETvalue to form the output image.
 16. The computing device of claim 15,wherein determining the TET sequence comprises determining a scenehistogram based on at least one of the images in the first plurality ofimages of the scene.
 17. The computing device of claim 16, wherein thescene histogram is based on downsampling and combining the images in thefirst plurality of images of the scene.
 18. The computing device ofclaim 16, wherein histograms are stored in a training database and arebased on at least two images captured of respective scenes, wherein thehistograms in the training database are associated with respectivedynamic range parameters, wherein the respective dynamic rangeparameters indicate whether the respective scenes exhibit low dynamicrange (LDR) or HDR, and wherein determining the TET sequence comprises:comparing the scene histogram to at least one histogram in the trainingdatabase; and based on an outcome of the comparison, determining adynamic range parameter for the scene from the respective dynamic rangeparameters, wherein determining the TET sequence is further based on thedynamic range parameter for the scene.
 19. The computing device of claim18, wherein the histograms in the training database are also associatedwith respective target pixel values, wherein determining the TETsequence further comprises: based on the respective target pixel valuesin the training database, determining one or more target pixel valuesfor the scene; and based on the one or more determined target pixelvalues, selecting one or more TET values to use in the TET sequence. 20.The computing device of claim 19, wherein the selected one or more TETvalues include the short TET value, the long TET value, and the fallbackTET value.